CS 545 : Assignment 8 Dan

Mixture of Probabilistic Principal Component Analyzers (MPPCA) is a seminal work in Machine Learning in that it was the first to use PCA to perform clustering and local dimensionality reduction. MPPCA is based upon the mixture of Factor Analyzers (MFA) which is similar to MPPCA except is uses Factor Analysis to estimate the covariance matrix. This algorithm is of interest to me because it is relevant to my Thesis work. Although MPPCA is presented as an improved variant of the mixture of Gaussians model, it may be better thought of as a form of covariance estimation. In covariance estimation, the covariance matrix is modified using a technique other than the standard XXT method of computing covariance of a data set where X = [x1, x2, . . . , xN ]. Furthermore MPPCA can be viewed as a form of covariance regularization which is a super-set of covariance estimation. Covariance regularization includes methods like shrinkage where the empirical covariance matrix is modified to be non-singular [7]. Also a part of covariance regularization is a common technique for computing a mixture of Gaussians (MoG) in high dimensions: constraining the covariance matrix to be diagonal. Historically, covariance regularization methods have been employed out of necessity where the training data is too few to compute a non-singular covariance matrix. In my Thesis, however, we see that covariance regularization techniques have a usefulness beyond making the data non-singular – that they can be beneficial to classification when used as a part of a MoG model and even when not necessary! An evaluation of MPPCA as a covariance regularization method would build upon my Thesis work in that it is a popular algorithm (with respect to the number of citations), derived by a respected figure in the machine learning community (and author of our course textbook), and is a form of MoG with covariance regularization. In addition my Thesis did not compare the performance of a covariance estimation, covariance regularization algorithm like MPPCA where the estimated covariance matrix is not merely modified, as with shrinkage, but is completely re-estimated using PCA. Personally, adding to the intrigue of this algorithm is that, despite its well-known status in the community, I have been unable to find a single, published result using this algorithm outside the publication that introduced it. Finally, an understanding of this algorithm’s performance and capabilities could be beneficial to, not only the machine learning community, but other communities which seek better techniques for analyzing data sets [6].